feat(orchestrator): carry the queue explicitly on every internal payload - #527
Merged
Conversation
behinddwalls
added a commit
that referenced
this pull request
Aug 6, 2026
…and writes (#525) ## Summary ### Why? `BatchStore.GetByQueueAndStates` was the storage contract's last query-by-attribute, backed by the schema's only secondary index (`idx_queue_state`). The queue's per-state membership records (`QueueBatchStateStore`) exist to replace it with pure by-key reads, but had no callers yet. ### What? Every batch state write now goes through the shared transition protocol in `submitqueue/core/batch`: `Transition` in the batch (Creating→Created), speculate (Speculating/Merging/Failed/Cancelled), mergesignal (terminal), cancel (Cancelling), and DLQ (Failed) controllers, with `EnsureRecord` repairing records after batch creation and on idempotent redelivery skip branches. The batch controller's dependency read goes through `core/batch.ListByStates` over the record buckets. `GetByQueueAndStates` is deleted from the contract, MySQL implementation, and mock, and `idx_queue_state` is dropped from the batch schema, so the batch table is pure get/put-by-key. Also repairs three stale controller tests that no longer matched the current control flow (they fail at the parent commit as well). ## Test Plan ✅ `go test ./...` (only pre-existing runway git-merger environment failures remain; they fail at the parent commit too) ✅ `make fmt` ✅ `make lint` ✅ `make gazelle` ✅ `make mocks` ✅ storage + orchestrator integration suites via `make integration-test` targets. ## Stack 1. @ #525 1. #527
behinddwalls
marked this pull request as ready for review
August 6, 2026 16:31
behinddwalls
force-pushed
the
preetam/queue-on-payloads
branch
from
August 6, 2026 16:39
7756304 to
229a29a
Compare
This was referenced Aug 6, 2026
## Summary ### Why? Consumers today learn a message's queue only by loading the referenced entity from storage, and entity IDs are opaque — nothing may parse a queue out of an ID prefix. For storage to become queue-scoped (resolved per queue like every other extension), every consumer must hold the queue before its first storage read, so the queue has to ride on the wire explicitly. ### What? The internal payload types (`RequestID`, `BatchID`, `BuildID`, `CancelRequest`) gain a `queue` field; the change is additive JSON, and payloads written before the field existed decode with an empty queue. Publishers stamp it from the entity they already hold: start, mergeconflictsignal, batch, speculate, mergesignal, build, buildsignal, and the orchestrator cancel controller; the gateway cancel controller stamps the authoritative queue from the stored request summary, overriding caller input. Consumers guard that a non-empty payload queue matches the loaded entity's queue and reject mismatches as malformed (non-retryable, straight to DLQ) — the guard becomes the routing input once storage is queue-resolved. Buildsignal's re-publish to speculate now partitions by the batch's queue instead of the inherited batch-ID partition key, matching every other speculate publisher and restoring the per-queue serial-processing guarantee that stage relies on. ## Test Plan ✅ `go test ./...` ✅ `make fmt` ✅ `make lint` — new unit tests cover the queue stamp on published payloads and the mismatch rejection guard.
behinddwalls
force-pushed
the
preetam/queue-on-payloads
branch
from
August 6, 2026 16:46
229a29a to
71c44ed
Compare
behinddwalls
marked this pull request as draft
August 6, 2026 16:46
behinddwalls
marked this pull request as ready for review
August 6, 2026 16:52
behinddwalls
marked this pull request as draft
August 6, 2026 16:56
behinddwalls
marked this pull request as ready for review
August 6, 2026 17:00
mnoah1
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
Consumers today learn a message's queue only by loading the referenced entity from storage, and entity IDs are opaque — nothing may parse a queue out of an ID prefix. For storage to become queue-scoped (resolved per queue like every other extension), every consumer must hold the queue before its first storage read, so the queue has to ride on the wire explicitly.
What?
The internal payload types (
RequestID,BatchID,BuildID,CancelRequest) gain aqueuefield; the change is additive JSON, and payloads written before the field existed decode with an empty queue. Publishers stamp it from the entity they already hold: start, mergeconflictsignal, batch, speculate, mergesignal, build, buildsignal, and the orchestrator cancel controller; the gateway cancel controller stamps the authoritative queue from the stored request summary, overriding caller input. Consumers guard that a non-empty payload queue matches the loaded entity's queue and reject mismatches as malformed (non-retryable, straight to DLQ) — the guard becomes the routing input once storage is queue-resolved. Buildsignal's re-publish to speculate now partitions by the batch's queue instead of the inherited batch-ID partition key, matching every other speculate publisher and restoring the per-queue serial-processing guarantee that stage relies on.Test Plan
✅
go test ./...✅make fmt✅make lint— new unit tests cover the queue stamp on published payloads and the mismatch rejection guard.